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We present a search for pair production of a fourth generation t' quark and its antiparticle, 
followed by their decays to a W boson and a jet, based on an integrated luminosity of 5.3 fb _1 of 
proton-antiproton collisions at y/s — 1.96 TeV collected by the DO Collaboration at the Fermilab 
Tevatron Collider. We set upper limits on the t't' production cross section that exclude at the 
95% C.L. a t' quark that decays exclusively to W+jet with a mass below 285 GeV. We observe a 
small excess in the ^i+jets channel which reduces the mass range excluded compared to the expected 
limit of 320 GeV in the absence of a signal. 

PACS numbers: 14.65.Jk, 13.85.Rm 



Measurements of the partial width of the Z boson to 
invisible final states at LEP exclude the existence of a 
fourth neutrino flavor with a mass less than half the Z 
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boson mass PQ. However, this does not exclude the ex- 
istence of a fourth generation of fermions as long as its 
neutrino is more massive. Precision electroweak data fa- 
vor a small mass splitting between the up-type quark of 
this fourth generation, t', and its down- type partner, b' , 
so that m(t') — m(b') < m(W) [\. Provided there is mod- 
erate mixing between the new fourth generation and the 
first three generations, the t' quark will predominantly 
decay to Wq, where q includes all standard model down- 
type quarks. 

We report on a search for a fourth generation t' quark 
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that is produced in proton-antiproton collisions together 
with its antiparticle. We assume that the t' quark is a 
narrow state that always decays to Wq. This search is 
also sensitive to other new particles that are pair pro- 
duced and decay to a W boson plus a jet. We select lep- 
ton+jets final states with one isolated electron or muon 
with high transverse momentum (pt)> a large imbalance 
in transverse momentum (pr), and at least four jets cor- 
responding to events in which one of the W bosons decays 
to leptons and the other W boson decays to quarks. A 
similar search has been carried out by the CDF Collab- 
oration in 0.8 fb _1 of integrated luminosity [3]. 

The DO detector consists of central tracking, calorime- 
ter, and muon systems [H |S] . The central tracking sys- 
tem is located inside a 2 T superconducting solenoidal 
magnet. Central and forward preshower detectors are lo- 
cated just outside of the coil and in front of the calorime- 
ters. The liquid-argon/uranium calorimeter is divided 
into a central section covering pseudorapidity |^| < 1.1 
and two end calorimeters extending 77 coverage to 4.2. 
The calorimeter is segmented longitudinally into elec- 
tromagnetic, fine hadronic, and coarse hadronic sections 
with increasingly coarser sampling. The muon system, 
located outside the calorimeter, consists of one layer of 
tracking detectors and scintillation trigger counters inside 
1.8 T toroidal magnets and two similar layers outside the 
toroids. A three-level trigger system selects events that 
are recorded for offline analysis. 

This analysis is based on data corresponding to an inte- 
grated luminosity of 5.3 fb _1 , collected by the DO Collab- 
oration at the Fermilab Tevatron proton-antiproton col- 
lider at a center of mass energy of y/s — 1.96 TeV. Events 
must satisfy one of several trigger conditions, all requir- 
ing an electron or muon with high transverse momentum, 
in some cases in conjunction with one or more jets. For 
all events, the pp collision point must be reconstructed 
with at least three tracks and located within 60 cm of the 
center of the detector along the beam direction. Jets are 
reconstructed using a midpoint cone algorithm [B] with 
cone size AR = y/(Arj) 2 + (Ac/)) 2 = 0.5, where <p is the 
azimuth, and must have at least two reconstructed tracks 
within the jet cone. The jet energy is corrected on av- 
erage to the total energy of all particles emitted inside 
the jet cone. Jets in simulated events are adjusted to 
reproduce the reconstruction efficiency and energy reso- 
lution and response observed in data. All events must 
have at least four jets with \rj\ < 2.5, pr > 40 GeV for 
the leading jet, and pr > 20 GeV for all other jets. The 
momentum carried away by neutrinos is inferred from the 
pr, computed from the energies in the cells of the elec- 
tromagnetic and fine hadronic calorimeters and adjusted 
for the energy corrections applied to the reconstructed 
jets and electrons and for the momentum of any recon- 
structed muons, taking into account their energy loss in 
the calorimeter. 

Electrons are identified as clusters of energy depo- 



sitions in the calorimeter that are isolated from other 
energy deposits. The electromagnetic section of the 
calorimeter must contain 90% of their energy, and the 
energy deposition pattern must be consistent with that 
of an electromagnetic shower. Every electron must be 
matched to a reconstructed track with pr > 5 GeV. For 
the e+jets channel, we require exactly one electron with 
Pt > 20 GeV and |f?| < 1.1 that originates from the 
pp collision point. We also require pr > 20 GeV and 
|A0(e,y T )| > 2.2 - 0.045 -pr/GeV, where A<j){e,p T ) is 
the azimuthual angle between electron andj/r, to reject 
events with jets that are misidentified as electrons. 

Muons arc defined as tracks reconstructed in the muon 
system matched to tracks in the central tracker. Muons 
must be separated from jets and isolated in the calorime- 
ter and in the tracker. For the /x+jets channel, we re- 
quire exactly one muon with px > 20 GeV and \r)\ < 2 
that originates from the pp collision point. The invari- 
ant mass of the selected muon and any other muon must 
be less than 70 GeV or more than 110 GeV to reject 
Z(— > ju/x)+jets events. We require j/t> 25 GeV and 
\A(j>{n^ T )\ > 2.1 - 0.035 -j/r/GeV to reject events with 
mismeasured muons. More details about the lepton+jets 
event selection can be found in Ref. [7J. 

The two main standard model processes that produce 
events with an isolated lepton, pr, and at least four jets 
are it and W+jets production. The third most impor- 
tant source of events arises from mismeasured multijet 
events in which a jet is misidentified as an electron or 
a muon from heavy flavor decay appears isolated. Sin- 
gle top quark, Z+jets, and diboson production can also 
give rise to such final states but have much smaller cross 
sections and/or acceptances. 

We use ALPGEN [5) to simulate it production with the 
top quark mass set to 172.5 GeV and generate additional 
jets from parton showers with PYTHIA [9]. We normalize 
the it sample to the theoretical it production cross sec- 
tion of 7.48^0.72 P b CD]- Samples of W+jets events are 
generated using ALPGEN and pythia with a jet-matching 
algorithm, following the MLM prescription [TT]. Three 
subsamples are generated: Wbb, Wcc, and VF+light par- 
tons. The Wc subprocesses are included in the W+light 
parton sample with massless charm quarks. We fix the 
relative normalization of Wbb, Wcc, and VF+light parton 
events to match NLO cross sections (T^]. The Z(-+ ee, 
fXfj,, Tr)+jets samples are generated with ALPGEN and 
pythia and broken up into Zbb, Zee, and Z+light parton 
samples in the same way as the Vy+jets samples. We fix 
their relative normalization to NLO predictions and nor- 
malize the total Z boson sample to the NNLO cross sec- 
tion [TS] . We simulate single top quark production using 
the COMPHEP-SINGLETOP [14] Monte Carlo event gener- 
ator with the top quark mass set to 172.5 GeV and nor- 
malize to the NNLO cross section with NNNLO threshold 
corrections in the s and ^-channels of 3.3 pb [TS]. Dibo- 
son samples are generated with pythia. Their NLO cross 
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sections are 12.3 pb for WW, 3.7 pb for WZ, and 1.4 pb 
for ZZ production [12]. The CTEQ6L1 parton distribu- 
tion functions [16] are used for all Monte Carlo samples. 
We simulate detector effects using the GEANT [T7] pro- 
gram. Events from random collisions are added to all 
simulated events to account for detector noise and addi- 
tional pp interactions. The events are reconstructed with 
the same program as the data. 

To define the background model, we estimate the num- 
ber of multijet events that enter the final data sample 
using a data driven method [IS]. We compute the num- 
ber of multijet events in the e+jets and /x+jets samples 
separately. We then subtract the multijet and all other 
backgrounds, except the W+jets background, from the 
data, based on their calculated cross sections, and nor- 
malize the VF+jets contribution to the remaining number 
of events. This corresponds to scaling the total number 
of W+jets events expected by a factor 1.3, which is con- 
sistent with NLO expectations. Table [I] summarizes the 
resulting composition of the data sample. When we test 
for the presence of a t' quark signal, we fix the relative 
normalizations of the IT+jets, Z+jets, single top quark, 
and diboson backgrounds, as given in Table [I] but float 
their overall normalization. 



TABLE I: Composition of the final data sample with system- 
atic uncertainties. The number of W+jets events is chosen to 
equalize the total number of events observed and expected. 



Source 


e+jets 


/i+jets 


tt production 


678+76 


508+55 


Single t production 


12+4 


8+3 


W+jets 


503+87 


648+59 


Z+jets 


41+7 


40+7 


WW, WZ, ZZ+sets 


25+5 


21+5 


Multijets 


173+42 


43+18 


Data 


1431 


1268 



To simulate the signal, we use t't' production in 
PYTHIA and force the decay t' — > Wb. However, since 
we do not identify b jets in this analysis, our results are 
also applicable to tf quarks decaying to a W boson and 
a light down-type quark. We generate events at 13 t' - 
mass values between 200 and 500 GeV. We set the total 
width of the t' quark to 10 GeV. This is smaller than the 
resolution for reconstructing the tf mass, which ranges 
between 50 GeV at m t < = 200 GeV and 100 GeV at 
m t > = 500 GeV. Therefore, the exact value of the width 
does not affect the analysis. 

We define Ht as the scalar sum of pr and of the trans- 
verse momenta of all jets and the charged lepton. A kine- 
matic fit to the t't' — > tvbqq'b hypothesis reconstructs the 
mass TOfit of the t' quark. We use the two-dimensional 
histograms of Ht versus mg t to test for the presence of 
signal in the data and to compute 95% C.L. upper limits 
on the t't' production cross section as a function of t' - 



mass. Figure [T] shows the scatter plots observed in data 
and expected from t't! production, tt production, and 
from all other background sources. For each hypothe- 
sized value of the t' mass, we fit the data to background- 
only and to signal+background hypotheses. We then use 
the likelihood ratio L = — 2 \og(Ps+B I Pb) as the test 
statistic, where Ps+b is the Poisson likelihood to observe 
the data under the signal+background hypothesis and 
Pb is the Poisson likelihood to observe the data under 
the background-only hypothesis. For the background- 
only hypothesis, we fit three components to the data: tt 
production constrained to its theoretical cross section, 
the multijets background constrained to the number of 
events given in Tableland IF+jets and all other back- 
grounds in the proportions given in Table [I] For the sig- 
nal+background fit we add the t't' cross section as a 
parameter to the fit. The fit can discriminate between 
background and signal contributions because their distri- 
butions in the Ht and mm variables are different. For 
each hypothesis we also vary the systematic uncertain- 
ties given in Table [TT] subject to a Gaussian constraint to 
their prior values to maximize the likelihood ratio [19) . 
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FIG. 1: Ht versus mg t for (a) data, (b) tt-production, (c) 
other background, and (d) t't' signal with m(t') — 325 GeV. 
The bins at the upper and right edges of the plots also contain 
overflows. 

We use the CL S method [2D] to determine the cross 
section limits. Using pseudoexperiments, we determine 
the probability to measure values of L that are larger 
than the value observed in the data sample for a t' signal, 
CL s+ b, and for no t' signal, CL^. The value of the t' pair 
production cross section for which l — CL s+ b/CLi, = 0.95 
is the 95% C.L. upper limit. We repeat this procedure 
for each t' mass point. 

Table [II] summarizes the sources of systematic uncer- 
tainties included in the limit calculation. The first four 
uncertainties affect the normalization of the components 
of our signal and background models. All other uncer- 
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tainties affect the selection efficiency. When estimating 
the effect of uncertainties in the jet energy scale, the jet 
identification efficiency, and the jet energy resolution, we 
also vary the shapes of the Hj> and mgt distributions. 
No uncertainties are given for the W+jets background 
because its normalization is a free parameter of the fit. 



TABLE II: Summary of systematic uncertainties above 1%. 
Some values vary with channel and with the time at which 
the data were taken. The numbers give the range for the size 
of the uncertainties. 



Source 


t'F 


tt 


multijets 


tt cross section 




9% 




Multijets normalization 






(25-50)% 


Integrated luminosity 


6.1% 


6.1% 




MC model 




4.3% 




Trigger efficiency 


<5% 


<5% 




pp collision point reconstruction 


1.6% 


1.6% 




Lepton identification 


(3-4)% 


(3-4)% 




Jet energy calibration 


(1-2)% 


(2-5)% 




Jet energy resolution 


(1-2)% 


(2-3)% 




Jet identification 


1% 


(1-3)% 
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FIG. 2: Distributions of (a) Ht and (b) mfl t for e+jets data 
and (c) Ht and (d) mat for /i+jets data compared with ex- 
pectations. The W/Z+jets category also includes single top 
quark and diboson production. The t't' signal is normalized 
to the expected yield. The unfilled histograms in (c) and (d) 
show the distributions with the best fit t'/T'-production cross 
section. 



We first analyze the e+jets and /x+jets data separately. 
Figure [2] shows the distributions of Ht and mg t from the 
standard model backgrounds and a 325 GeV t' quark 
signal compared to data. There is no visible excess in 
the e+jets data. In the /Lt+jets data we observe a small 
excess of events over standard model expectations. We 
can fit the data best with a t't' production cross section 
of 3.2 ±1.1 times the theoretical cross section for a t' 
quark mass of 325 GeV. The value of 1 — CLf, for the 
data gives the probability of getting a local deviation of 
at least this size from the standard model expectation in 
the absence of physics beyond the standard model. Wc 
find a p value of 0.007, corresponding to 2.5 Gaussian- 
equivalent standard deviations. 

Figure [3] shows the resulting cross section limits com- 
pared to the limits expected in the absence of t't' produc- 
tion and to the predicted NLO t' pair production cross 
section [2T] as a function of the t' mass. We expect to 
be able to exclude t't' production for t' quark masses be- 
low 315 GeV in the e+jets channel and below 280 GeV 
in the /z+jets channel. The observed cross section limit 
allows us to exclude t't' production for t' quark masses 
at the 95% C.L. below 295 GeV in the e+jets channel 
and below 225 GeV in the /x+jets channel. Combining 
e+jets and /i+jets data as shown in Fig. |1J we expect 
to exclude t't' production for t' quark mass values below 
320 GeV. Based on the observed limits we can exclude at 
the 95% C.L. t't' production for t! quark masses below 
285 GeV. We achieve the best fit to the data with a t't' 
production cross section of 1.1 ± 0.5 times the theoreti- 
cal cross section for a t' quark mass of 325 GeV which 
gives a p value of 0.015, corresponding to 2.2 standard 
deviations from zero. 




f mass [GeV] f mass [GeV] 



FIG. 3: Observed and expected upper limits and predicted 
values for the t't' production cross section as a function of the 
mass of the t' quark for (a) e+jets, (b) /^+jets. The shaded 
regions around the expected limit represent the ±1 and ±2 
standard deviation bands. 

In conclusion, we searched for pair production of a t! 
quark and its antiparticle followed by their decays into a 
W boson and a jet. We do not see a signal consistent with 
t't' production, although we observe a small excess of 
events in the ^t+jets channel. Combining the e+jets and 
/i+jets channels, we exclude at 95% C.L. t't' production 
for t' quark mass values below 285 GeV. 
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FIG. 4: Same as Fig. [3] but for both channels combined. 
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